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Student Outcome Survey: self-reported graduate model review 


Introduction 


The National Student Outcomes Survey (SOS) collects information about students who 
completed their vocational education and training (VET) in the previous calendar year. 
The gathered information on the surveyed VET students includes their reasons for 
training, employment outcomes, satisfaction with training, and further study outcomes. 
The survey covers students who have completed a qualification (graduates) and those 
who have completed only part of a course and then left the VET system (subject 
completers). The National Centre for Vocational Education Research (NCVER) has 
conducted the survey with government-funded VET students annually since 1999. In 
2016, the scope of the survey was expanded to report on the outcomes of graduates 
whose training was Commonwealth— or state-funded as well as fee-for-service 
graduates. These graduates were referred to as total VET graduates. An explanation of 
the difference between total VET and government-funded student outcomes can be 
found in appendix A. The expanded scope was applied to the 2017 survey for graduates 
(following a successful trial in 2016) and for the first time for subject completers and the 
series renamed VET student outcomes. 


At the time of sample selection, insufficient information is available from the National 
VET Provider Collection to identify ‘actual’ subject completers. Instead, a sample of 
potential subject completers is chosen, which includes students who are continuing in 
the VET system. The status of respondents is determined through the survey responses. 
As such, respondents to the SOS include a number of students who were sampled as 
subj ect completers based on administrative data reported to the National VET Provider 
Collection, but self-identify in the questionnaire as graduates. For many years these 
‘self-reported graduates’ (SRGs) were categorised as graduates in survey outputs, 
because the self-report was deemed to be more reliable than the collection data. 
However, it became apparent that many SRGs were not, in fact, graduates. In response 
to this issue, in 2012, NCVER created a logistic model that predicted the eligibility of a 
SRG being an ‘actual’ graduate based on their personal and training characteristics 
(Braysher 2012). This model has since been run annually for each SOS to assign group 
membership to SRGs. Those SRGs that were not predicted by the model to be a graduate 
were re-assigned to their original subject completer status. 


One of the conditions of the initial analysis was that the model should be reviewed at 
least every four years to assess its ongoing validity and to make possible modifications 
should demographics and administrative data change and alter the predictive power of 
the model. The model was reviewed by NCVER in 2015 (unpublished). The review found 
some changes in data quality, but found no evidence that these changes were affecting 
the estimates. At the time no changes were recommended to the logistic model or 
graduate reclassification procedure, but it was recommended that the model be 
reviewed again at a later stage. In relation to data quality, the proportion of subject 
completers claiming to be self-reported graduates has declined significantly since 2007, 
particularly from 2015 to 2017, highlighting the improvement in the quality of the 
National VET Provider Collection data since the need to run the model arose (appendix 
B). Hence, it was appropriate to review the model again. This report presents the 
findings of an additional comprehensive review of the model and methodology. 
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Methodology 


Determining the status of self-reported graduates (SRGs) 


By definition, a Student Outcomes Survey (SOS) graduate must have completed their 
training in the previous calendar year to that of the survey. For example, to be an 
eligible graduate for the 2017 SOS, the student must have completed their training and 
been awarded a qualification in 2016. Furthermore, the qualification they received must 
be the same as that recorded in the National VET Provider Collection, which is used as 
the survey sampling frame. 


In the process of creating a logistic model to predict the status of SRGs, it is necessary to 
determine the ‘true’ status of SRGs. For the analysis presented in this report, the set of 
rules/ conditions used was based on data from three National VET Provider Collections 
for each SOS. Under the original methodology, students were considered eligible 
graduates if: 


A. in any of the three relevant collections (corresponding to the source year and the 
two subsequent years) they were recorded as having completed a qualification in the 
source year 


B. the qualification they received in the source year was the same as that for which 
they were sampled 


C. they were recorded as having completed a qualification in the first year subsequent 
to the source year, and this qualification was the same as that for which they were 
sampled. 


Note that SRGs meeting condition C are not graduates according to the SOS definition 
because they completed their training in the year of the survey rather than in the 
previous year. In this report we also consider using an alternative rule to classify SRGs as 
graduates: a SRG is considered to be a graduate if they meet conditions A and B above. 
This distinction has a significant effect on the predictive power of the logistic model 
when classifying self-reported graduates and also on some of the key survey measures 
from the SOS, which will be highlighted in this report. 


To fit the logistic model, a data set that contained every SRG for each SOS from years 
2007-09 was created that contained their personal and training characteristics, as well 
as their responses to the SOS. Using the relevant VET Provider Collections and rules 
stated previously, a variable was added to this data set that indicated whether a SRG 
was indeed eligible to be considered a graduate. This variable, representing a student’s 
eligibility, was treated as the binary response variable for the logistic model. For each 
SOS, the model fitted to the data set of 200709 SRGs is used to determine the 
probability of being a graduate for each of the students in the SOS of interest. An 
implication of this method is that we are assuming the SRGs from 2007-09 are 
representative of the SRG population for the SOS year of interest. It is conceivable that 
the characteristics that make up a SRG in 2017 or any other year could be different to 
that of those that made up one in 2007-09. Hence, this assumption could have 
significant implications and may not be appropriate. 
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Options considered 


In order to determine the best strategy going forward for group classification for those 
students who self-report as graduates in the SOS, five methods were considered and 
investigated: 


1. Continue using the current method (logistic model based on 2007-09 data) and 
eligibility rules as developed by NCVER (A+B+C). 


2. Continue using the current method (logistic model based on 2007-09 data) but with 
the updated eligibility rules (A+B). 


3. Use a logistic model based on 2012—14 data with the updated eligibility rules. 


4. Keep the status of all SRGs as they are at sampling, i.e. subject completers. It must 
be noted that after these students are kept as subject completers, they are then run 
through a process using their survey responses to determine whether they are 
continuing in the VET sector or not. Continuing students are not included in outputs 
published by NCVER. 


5. Noting quarterly reporting is only available for government-funded data, for non- 
government funded records, keep the status of SRGs as they are at sampling - 
subject completers. For government-funded data, change the status of a SRG from 
subject completer to graduate if they are reported as an eligible graduate in the 
quarterly VET data collection from the first quarter of the year following the source 
year; otherwise, leave the status of SRGs as they are at sampling - subject 
completers. 


Note that options 1, 2 and 3 all employ a logistic model with the original predictor 
variables of the model. 
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Results 


Evaluating the options 


To investigate the accuracy of each of the options, the percentage of correctly classified 
self-reported graduates for the 2009, 2015 and 2016 surveys, as well as the difference 
between estimates of key variables produced using the ‘exact’ classification of SRGs and 
the classification using the option was calculated. The ‘exact’ group that each SRG 
belonged to was determined using the relevant VET Provider Collections. For each SOS, 
three collections were used to govern whether a student was an eligible graduate. The 
collections used were: the collection from which the survey sample was selected (the 
source collection) and the two subsequent collections. To be considered an ‘exact’ 
graduate, a student must have completed their training in the previous calendar year to 
that of the survey, and that the recorded completion was the same program as the one 
they were sampled for in the SOS. This is consistent with how a graduate is defined for 
the SOS (using eligibility rules A+B). Using this ‘exact’ classification, the percentage of 
correctly classified SRGs can be determined. 


Each of the five options considered were employed for the 2015 and 2016 SOS, and their 
correct classification percentage’ is illustrated in figure 1 and figure 2. 


Figure 1 Correct classification under the five options for determining group membership of 


SRGs for the 2015 SOS 
100.0 » 929 
90.0 - 825 85.4 85.7 
80.0 - 
70.0 | 67.6 
60.0 + 
% 500 - 
40.0 - 
30.0 
20.0 
10.0 
0.0 + 


Options 


= One - current method 

= Two - current method plus updated eligibility rules 

= Three - logistic model based on 2012-14 data with updated eligibility rules 
= Four - keep status of SRGs as they are at sampling 


Five - make use of quarterly reporting for govemment-funded data 


1 Results are based on government-funded data using the scope defined in 2016. This means that TAFE 
fee-for-service students are included in the government-funded scope, which is not the case for the 
scope defined in 2017. 
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Figure 2 Correct classification under the five options for determining group membership of 
SRGs for the 2016 SOS 
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Options 


mOne - current method 

m= Two - current method plus updated eligibility rules 

= Three - logistic model based on 2012-14 data with updated eligibility rules 
m= Four - keep status of SRGs as they are at sampling 


Five - make use of quarterly reporting for government-funded data 


A comparison was then made as to whether using the model was more appropriate than 
just leaving the SRGs as they were at the time of sampling - subject completers. This 
was done by analysing how well the two methods predicted some of the key survey 
measures for the SOS in the chosen years. In this context, results for the key survey 
measures under the two methods were considered ‘accurate’ if there was no significant 
difference between them and the ‘exact’ results. It was observed that leaving the SRGs 
as subject completers will produce results for the key survey measures that are as good 
as, if not better than, the results that are calculated when the model is used (refer to 
appendix C). To validate the assumption, that leaving the SRGs as subject completers is 
more appropriate than using the model, the percentage of correct classifications was 
found to be higher in each of the chosen surveys when the SRGs were left as subject 
completers (refer to appendix D). However, it must be noted that if the methodology is 
changed, and the original rules (A+B+4C) for eligibility are applied to determine the 
‘exact’ group each student belongs to, then it is seen that the model performs slightly 
better (in terms of correct classification percentage) than leaving the students as 
subject completers (refer to appendix D). 
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Conclusion 


When analysing figures 1 and 2, it is noted that option five yields the best results in 
terms of correctly classifying the SRGs into their group. However, total VET activity 
(TVA) data are not reported quarterly, and so this option would not allow a consistent 
method of classifying SRGs across the SOS. The next best option in terms of correctly 
classifying the SRGs into their group is option four, which enables a consistent approach 
in determining what group a student belongs to in both government-funded and TVA 
data. Discontinuing the use of the model can also be justified by observing how the 
number of SRGs over the past ten years has seen a decline. There has also been a 
noticeable decrease in the percentage of SRGs out of the total number of subj ect 
completers at sampling since 2015 (a decrease from 33.2% in 2015 to 12.1%in 2017) 
(refer to appendix B). These results coincide with quarterly reporting for government- 
funded data being introduced in the 2015 survey (2014 National VET Provider Collection). 
With this quarterly reporting, data quality has likely been improved, and in turn the 
impact of lags in reporting have diminished, resulting in less SRGs in the SOS. This gives 
further evidence that abandoning the use of the model now may be the most appropriate 
time to do it. 


In final, the recommendation going forward is to employ option four. That is, keeping 
the SRGs as they are at the time of sampling - subject completers. However, if at some 
point in the future TVA data are reported quarterly as well, then utilising option five 
may be the best and most suitable approach to classifying the group status of self- 
reported graduates. 


For the 2017 SOS, option four was used to classify self-reported graduates. NCVER back 
casted data for the 2016 survey, employing option four as the classification method for 
SRGs. All surveys prior to, and including 2015 were not back casted, as the need to 
utilise this new methodology was not required during this period. 
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Appendix A 


Difference between total VET and government-funded 
student outcomes 


The National Student Outcomes Survey is undertaken as a stratified, randomly selected 
sample from the National VET Provider Collection. In 2016, the National Centre for 
Vocational Education Research released two publications containing data from the 
National Student Outcomes Survey: 


= Government-funded student outcomes 
« Total VET graduate outcomes. 


There are distinct differences in the data reported under these two scopes. 


Government-funded student outcomes 


Government-funded student outcomes provides information on the outcomes of students 
who completed government-funded VET in Australia. In 2016 this was broadly defined as 
all activity delivered by government providers and government-funded activity delivered 
by community education and private training providers (see table Al). 


Table Al Government-funded student outcomes scope matrix 


TAFE and other Community Private training 
government providers’ education providers providers 
Commonwealth and state 
funding v v v 
Domestic fee-for-service” v x x 
International fee-for-service x x x 


Note: 1. In the Government-funded student outcomes publication, universities are reported as ‘TAFE and other 
government providers’. 


2. In 2017 the government-funded scope was changed to not include domestic fee-for-service students. 
Results are based on the 2016 scope. 


Total VET graduate outcomes 


Total VET graduate outcomes provide information on the outcomes of graduates who 
completed their VET in Australia. It includes the outcomes of graduates in receipt of 
Commonwealth— or state—funding, as well as those who paid for their training (see table 
A2). The mandatory reporting of nationally recognised training activity to the National 
VET Provider Collection provided a sampling frame to expand the scope of the National 
Student Outcomes Survey to include fee-for-service graduates from private training and 
community education providers. Fee-for-service graduates are those who paid for the 
training or whose employer paid for the training. 
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Table A2 Total VET graduate outcomes scope matrix 


TAFE and other Community Private training 
government providers education providers providers 
Commonwealth and state 
funding v ¥ v 
Domestic fee-for-service v Y Y 
International fee-for-service x x x 


In 2017, total VET subject completers have been included and the series has been 
renamed VET student outcomes. 
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Appendix B 


Summary of VET data quality, 2007-17 


Since 2010, the total number of self-reported graduates in each SOS has followed a 
decreasing trend (figures B1 and B2). To allow for years in this time series to be 
comparable, they were split into ‘big’ and ‘small’ SOS years.” * 4 


Figure B1 Number of SRGs for ‘big’ SOS years, 2007-17 
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Figure B2 Number of SRGs for ‘small’ SOS years, 2008-16 
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2 Historically the survey was conducted to allow TAFE institute level reporting for graduates and subject 
completers every two years (big survey year), with national and state level reporting carried out every 
year. 

3 In 2017, 15—17 year old students were excluded from the survey scope. 

4 Results for each SOS from 2007—16 are based on government-funded data using the 2016 defined 
government-funded scope. Results for the 2017 SOS are generated from total VET activity (TVA) scope, 
as this was the first year subj ect completers were included in the TVA scope. 
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Improvements in data quality have a direct impact on group classification for students in 
the SOS, specifically the self-reported graduates. Table B1 displays how the percentage 
of SOS self-reported graduates out of the total number of subject completers at sampling 
has dropped dramatically since 2015. This is likely attributed to the introduction in 2015 
of quarterly reporting for government-funded data. 


Table B1 Percentage of self-reported graduates out of all subject completers at sampling, 


2007-17 
Year Per cent (%) 
2007 29.1 
2008 29.2 
2009 30.1 
2010 31.4 
2011 32.8 
2012 35.7 
2013 33.8 
2014 33.9 
2015 33.2 
2016 24.5 
2017 12.1 
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Appendix C 


Results for key survey measures under two different methods 


Tables C1—C6 display key survey measures for the 2009, 2015 and 2016 SOS under two 
methods (the model, and leaving the SRGs as subj ect completers) and comparing them 
with the ‘exact’ results for the estimates. ° The highlighted cells in each table represent 
a statistically significant difference between the estimate when SRGs are classified using 
VET Provider Collection data (labelled Exact), and the estimate calculated when SRGs 
are classified under the method used. Statistical significance was assessed using a two- 


sample t-test with significance level 0.05. 


Table C1 Comparison of government-funded SOS key measures using the model and the 


exact results to classify SRGs for the SOS, 2009 


WET ce T Ta) 
of 


WM ETaetia 


oi 


Significance 


Exact Model like) g error testing 

Graduates 
Employed after training 78.9 79.7 0.5 0.5 Slemelis 
Employed or in further study after training 89.0 89.6 0.4 0.4 
Enrolled in further study after training 35.4 35.0 0.6 0.6 similar 
Achieved main reason for doing the training 85.8 86.1 0.4 0.4 similar 
Satisfied with the overall quality of training 88.7 88.5 0.4 0.4 similar 
Of those employed after training 

Found the training relevant to their current job 79.3 79.7 0.6 0.5 similar 

Received at least one job-related benefit 75.3 75.8 0.6 0.6 similar 
Of those not employed before training 

Employed after training 46.5 47.8 1.3 1.2 similar 
Of those employed before training 

Employed after training at a higher skill level 24.5 25.1 0.7 0.7 similar 
Subject completers 
Employed after training 74.9 74.7 0.6 0.6 similar 
Employed or in further study after training 79.4 78.6 0.5 0.5 Sle melis 
Enrolled in further study after training 14.6 8.9 0.4 0.3 
Achieved main reason for doing the training 83.8 83.7 0.5 0.5 similar 
Satisfied with the overall quality of training 86.2 86.2 0.5 0.5 similar 
Of those employed after training 

Found the training relevant to their current job 68.7 68.1 0.7 0.7 similar 

Received at least one job-related benefit 60.0 58.8 0.7 0.8 
Of those not employed before training 

Employed after training 31.6 30.4 1.2 1.3 similar 
Of those employed before training 

Employed after training at a higher skill level 12.5 10.9 0.5 0.5 


5 The estimates in tables Cl1—C6 are calculated under the 2016 defined government-funded scope 


(includes fee-for-service TAFE students). 
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Table C2 Comparison of government-funded SOS key measures using the status of self- 
reported graduates at sampling and the exact results to classify SRGs for the SOS, 


2009 
Margin Margin 
PX of of Significance 
=> Cle Mt Lin) e)(=L8 | error error testing 

Graduates 
Employed after training 78.9 78.5 0.5 0.5 similar 
Employed or in further study after training 89.0 89.2 0.4 0.4 similar 
Enrolled in further study after training 35.4 36.5 0.6 0.6 
Achieved main reason for doing the training 85.8 85.9 0.4 0.5 similar 
Satisfied with the overall quality of training 88.7 88.8 0.4 0.4 similar 
Of those employed after training 

Found the training relevant to their current job 79.3 79.0 0.6 0.6 similar 

Received at least one job-related benefit 75.3 74.9 0.6 0.6 similar 
Of those not employed before training 

Employed after training 46.5 46.0 1.3 1.3 similar 
Of those employed before training 

Employed after training at a higher skill level 24.5 24.1 0.7 0.7 similar 
Subject completers 
Employed after training 74.9 75.3 0.6 0.5 similar 
Employed or in further study after training 79.4 80.0 0.5 0.5 similar 
Enrolled in further study after training 14.6 15.6 0.4 0.4 
Achieved main reason for doing the training 83.8 83.9 0.5 0.5 similar 
Satisfied with the overall quality of training 86.2 86.4 0.5 0.4 similar 
Of those employed after training 

Found the training relevant to their current job 68.7 69.6 0.7 0.6 similar 

Received at least one job-related benefit 60.0 61.2 0.7 0.7 
Of those not employed before training 

Employed after training 31.6 32.7 1.2 1.2 similar 
Of those employed before training 

Employed after training at a higher skill level 12.5 13.5 0.5 0.5 
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Table C3. Comparison of government-funded SOS key measures using the model and the 


exact results to classify SRGs for the SOS, 2015 


Margin Margin 
of of Significance 
| > Clo; aero (| Claires laces testing 

Graduates 
Employed after training 73.4 74.2 0.8 0.8 similar 
Employed or in further study after training 84.6 85.2 0.7 0.6 similar 
Enrolled in further study after training 32.8 32.6 0.8 0.8 similar 
Achieved main reason for doing the training 79.7 80.4 0.7 0.7 similar 
Satisfied with the overall quality of training 86.8 86.7 0.6 0.6 similar 
Of those employed after training 

Found the training relevant to their current job 77.9 78.8 0.9 0.9 similar 

Received at least one job-related benefit 74.5 75.1 0.9 0.9 similar 
Of those not employed before training 

Employed after training 40.4 41.2 1.6 1.6 similar 
Of those employed before training 

Employed after training at a higher skill level 15.7 16.3 0.8 0.8 similar 
Subject completers 
Employed after training 71.6 71.3 0.8 0.9 similar 
Employed or in further study after training 76.4 75.4 0.8 0.9 similar 
Enrolled in further study after training 15.5 10.4 0.7 0.6 
Achieved main reason for doing the training 80.1 79.5 0.8 0.8 similar 
Satisfied with the overall quality of training 84.7 84.6 0.7 0.7 similar 
Of those employed after training 

Found the training relevant to their current job 71.7 70.3 1.0 1.0 similar 

Received at least one job-related benefit 64.0 62.6 1.0 1.1 similar 
Of those not employed before training 

Employed after training 32.2 32.0 1.7 1.8 similar 
Of those employed before training 

Employed after training at a higher skill level 8.9 7.7 0.7 0.7 
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Table C4 Comparison of government-funded SOS key measures using the status of self- 
reported graduates at sampling and the exact results to classify SRGs for the SOS, 


2015 
Margin Margin 
EX of of Significance 
Exact sampled error error testing 

Graduates 
Employed after training 73.4 73.4 0.8 0.9 similar 
Employed or in further study after training 84.6 84.7 0.7 0.7 similar 
Enrolled in further study after training 32.8 33.0 0.8 0.9 similar 
Achieved main reason for doing the training 79.7 79.7 0.7 0.8 similar 
Satisfied with the overall quality of training 86.8 86.9 0.6 0.7 similar 
Of those employed after training 

Found the training relevant to their current job 77.9 78.1 0.9 1.0 similar 

Received at least one job-related benefit 74.5 74.5 0.9 1.0 similar 
Of those not employed before training 

Employed after training 40.4 40.6 1.6 1.8 similar 
Of those employed before training 

Employed after training at a higher skill level 15.7 15.7 0.8 0.9 similar 
Subject completers 
Employed after training 71.6 71.8 0.8 0.8 similar 
Employed or in further study after training 76.4 77.0 0.8 0.8 similar 
Enrolled in further study after training 15.5 16.9 0.7 0.6 
Achieved main reason for doing the training 80.1 80.1 0.8 0.7 similar 
Satisfied with the overall quality of training 84.7 84.9 0.7 0.6 similar 
Of those employed after training 

Found the training relevant to their current job 71.7 72.1 1.0 0.9 similar 

Received at least one job-related benefit 64.0 65.1 1.0 1.0 similar 
Of those not employed before training 

Employed after training 32.2 32.8 1.7 1.6 similar 
Of those employed before training 

Employed after training at a higher skill level 8.9 9.7 0.7 0.6 similar 
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Table C5 Comparison of government-funded SOS key measures using the model and the 


exact results to classify SRGs for the SOS, 2016 


Margin Margin 
of of Significance 
Exact Model laces lake) g testing 

Graduates 
Employed after training 74.1 74.9 0.7 0.7 similar 
Employed or in further study after training 85.0 85.4 0.6 0.5 similar 
Enrolled in further study after training 32.2 32.3 0.7 0.7 similar 
Achieved main reason for doing the training 82.4 82.9 0.6 0.6 similar 
Satisfied with the overall quality of training 86.2 86.1 0.6 0.5 similar 
Of those employed after training 

Found the training relevant to their current job 76.8 77.9 0.8 0.8 similar 

Received at least one job-related benefit 67.5 68.0 0.9 0.8 similar 
Of those not employed before training 

Employed after training 43.3 43.8 1.4 1.4 similar 
Of those employed before training 

Employed after training at a higher skill level 18.5 18.2 0.8 0.8 similar 
Subject completers 
Employed after training 71.6 70.8 1.2 1.3 similar 
Employed or in further study after training 75.6 74.4 1.1 1.2 similar 
Enrolled in further study after training 11.2 9.8 0.9 0.9 
Achieved main reason for doing the training 79.4 78.5 1.1 1.2 similar 
Satisfied with the overall quality of training 82.6 82.3 1.0 41. similar 
Of those employed after training 

Found the training relevant to their current job 69.3 67.3 1.5 1.6 similar 

Received at least one job-related benefit 52.5 50.5 1.6 1.7 similar 
Of those not employed before training 

Employed after training 33.9 33.5 2.4 2:5 similar 
Of those employed before training 

Employed after training at a higher skill level 9.2 8.9 1.0 el similar 
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Table C6 Comparison of government-funded SOS key measures using the status of self- 


reported graduates at sampling and the exact results to classify SRGs for the SOS, 


2016 
Margin Margin 
As of of Significance 
Exact sampled error error testing 

Graduates 
Employed after training 74.1 74.1 0.7 0.7 similar 
Employed or in further study after training 85.0 85.2 0.6 0.6 similar 
Enrolled in further study after training 32.2 33.0 0.7 0.7 similar 
Achieved main reason for doing the training 82.4 82.5 0.6 0.6 similar 
Satisfied with the overall quality of training 86.2 86.2 0.6 0.5 similar 
Of those employed after training 

Found the training relevant to their current job 76.8 77.1 0.8 0.8 similar 

Received at least one job-related benefit 67.5 67.5 0.9 0.9 similar 
Of those not employed before training 

Employed after training 43.3 43.4 1.4 1.4 similar 
Of those employed before training 

Employed after training at a higher skill level 18.5 18.1 0.8 0.8 similar 
Subject completers 
Employed after training 71.6 71.7 1.2 1.1 similar 
Employed or in further study after training 75.6 75.9 1.1 1.1 similar 
Enrolled in further study after training 11.2 11.5 0.9 0.8 similar 
Achieved main reason for doing the training 79.4 79.5 1.1 1.1 similar 
Satisfied with the overall quality of training 82.6 82.8 1.0 1.0 similar 
Of those employed after training 

Found the training relevant to their current job 69.3 69.4 1.5 1.4 similar 

Received at least one job-related benefit 52.5 53.4 1.6 1.5 similar 
Of those not employed before training 

Employed after training 33.9 34.4 2.4 2.3 similar 
Of those employed before training 

Employed after training at a higher skill level 9.2 10.1 1.0 1.0 similar 


For each SOS, it is observed that there is an equal or less amount of significant 
differences when using the method whereby SRGs are left as subject completers 


compared to that of when the model is used. 


Student Outcome Survey: self-reported graduate model review 


Appendix D 


Correctly classifying SRGs into their group under the new 
methodology 


Under the new methodology (not allowing a student to be considered a graduate if they 
completed their training in the year they took the survey) for eligibility, the ‘exact’ 
group each student belongs to was determined. Three National VET Provider Collections 
were used in order to conclude the ‘exact’ group. These collections were the source 
collection and the two following collections. For example, for the 2009 survey, the 2008- 
10 National VET Provider Collections were used. The percentage of correctly classified 
self-reported graduates under the two methods, i.e. the percentage that matched the 
‘exact’ classification for the 2009, 2015 and 2016 SOS was determined and is displayed in 
figure D1. This was done to validate the assumption that leaving the SRGs as subj ect 
completers (option four) is more appropriate than using the model (option one), when 
the updated eligibility rules are used to determine the ‘exact’ group each student 
belongs to. 


Figure D1_ Percentage of correctly classified SOS self-reported graduates under new 
methodology, 2009, 2015 and 2016 


% 
3 


SOS 2009 SOS 2015 SOS 2016 


|= Model (option one) 78.1 707 676 


|= Remain as sampled (option four) 844 80.6 85.7 


Observing figure D1, it is seen that leaving the SRGs as subject completers yields a 
higher correct classification percentage for each SOS year that was analysed. 
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Correctly classifying SRGs into their group under the original 
methodology 


The ‘exact’ group each student belongs to was determined under the original 
methodology (allowing a student to be deemed an eligible graduate if they had 
completed their training in the year they took the survey) for the same SOS years (2009, 
2015 and 2016). The percentage of correctly classified self-reported graduates under the 
two methods was then determined and is displayed in Figure D2. 


Figure D2 Percentage of correctly classified SOS self-reported graduates under original 
methodology, 2009, 2015, and 2016 


100 


SOS 2009 SOS 2015 SOS 2016 


= Model (option one) 76.1 68.1 67.3 


m= Remain as sampled (option four) 73.9 66.5 67.2 


If the original rules for eligibility are applied to determine the ‘exact’ group each 
student belongs to, then it is seen that the model (option one) performs slightly better 
(in terms of correct classification percentage) than leaving the students as subj ect 
completers (option four). However, this is not appropriate as it does not align with the 
definition of a SOS graduate, which does not allow a student to be considered a graduate 
if they have completed their training in the same year as the survey. 
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